Actively adapting to driving environments based on human interactions

ABSTRACT

A method performed by an autonomous vehicle includes identifying a condition preventing the autonomous vehicle from proceeding along an intended route. The method also includes prompting a passenger of the autonomous vehicle to interact with a driver of a first vehicle in response to identifying the condition. The method further includes receiving, from the passenger, an input at an interface of the autonomous vehicle indicating a successful interaction or an unsuccessful interaction with the driver. The method also includes controlling the autonomous vehicle to proceed along the intended route based on the input indicating the successful interaction with the driver.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/279,857, filed on Feb. 19, 2019, and entitled “ACTIVELY ADAPTING TODRIVING ENVIRONMENTS BASED ON HUMAN INTERACTIONS,” the disclosure ofwhich is expressly incorporated by reference herein in its entirety.

BACKGROUND Field

Certain aspects of the present disclosure generally relate to autonomousvehicles and, more particularly, to a system and method for adjusting anaction of an autonomous vehicle in response to human interactions.

Background

In most cases, autonomous vehicles perform actions based on developedrules. For example, a set of rules may be used to safely navigate theautonomous vehicle on a multilane road. The set of rules may include aminimum following distance between vehicles in a single lane and also aminimal gap for merging between two vehicles. Actions of the autonomousvehicle are constrained by the set of rules. In theory, if theautonomous vehicle follows each rule, the autonomous vehicle should notbe the cause of an accident.

To prevent accidents, the rules may be conservative. Due to theconservative nature of the rules, in some situations, such as heavytraffic situations, the rules may increase driving times. For example,the rules may cause the autonomous vehicle to wait for an idealsituation to merge. It is desirable to improve autonomous vehicles tomaintain safety while also mitigating delays and/or other driverinconveniences that may result from following the rules.

SUMMARY

In one aspect of the present disclosure, a method for controlling anautonomous vehicle is disclosed. The method includes navigating theautonomous vehicle based on a set of rules. The method also includesidentifying an abnormality in a current driving situation. The methodfurther includes prompting a passenger of the autonomous vehicle tointeract with a driver of a first vehicle in response to identifying theabnormality. The method still further includes controlling theautonomous vehicle to violate one or more rules of the set of rules inresponse to an indication of a successful interaction with the driver.

In another aspect of the present disclosure, a non-transitorycomputer-readable medium with non-transitory program code recordedthereon is disclosed. The program code is for controlling an autonomousvehicle. The program code is executed by a processor and includesprogram code to navigate the autonomous vehicle based on a set of rules.The program code also includes program code to identify an abnormalityin a current driving situation. The program code further includesprogram code to prompt a passenger of the autonomous vehicle to interactwith a driver of a first vehicle in response to identifying theabnormality. The program code still further includes program code tocontrol the autonomous vehicle to violate one or more rules of the setof rules in response to an indication of a successful interaction withthe driver.

Another aspect of the present disclosure is directed to an apparatus forcontrolling an autonomous vehicle. The apparatus having a memory and oneor more processors coupled to the memory. The processor(s) is configuredto navigate the autonomous vehicle based on a set of rules. Theprocessor(s) is also configured to identify an abnormality in a currentdriving situation. The processor(s) is further configured to prompt apassenger of the autonomous vehicle to interact with a driver of a firstvehicle in response to identifying the abnormality. The processor(s)still further configured to control the autonomous vehicle to violateone or more rules of the set of rules in response to an indication of asuccessful interaction with the driver.

This has outlined, rather broadly, the features and technical advantagesof the present disclosure in order that the detailed description thatfollows may be better understood. Additional features and advantages ofthe present disclosure will be described below. It should be appreciatedby those skilled in the art that this present disclosure may be readilyutilized as a basis for modifying or designing other structures forcarrying out the same purposes of the present disclosure. It should alsobe realized by those skilled in the art that such equivalentconstructions do not depart from the teachings of the present disclosureas set forth in the appended claims. The novel features, which arebelieved to be characteristic of the present disclosure, both as to itsorganization and method of operation, together with further objects andadvantages, will be better understood from the following descriptionwhen considered in connection with the accompanying figures. It is to beexpressly understood, however, that each of the figures is provided forthe purpose of illustration and description only and is not intended asa definition of the limits of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, nature, and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify correspondingly throughout.

FIG. 1 illustrates an example of a vehicle in an environment accordingto aspects of the present disclosure.

FIG. 2A illustrates an example of an abnormal driving situationaccording to aspects of the present disclosure.

FIG. 2B illustrates an example of an interactive interface according toaspects of the present disclosure.

FIG. 2C illustrate an example of passenger and driver interactionaccording to aspects of the present disclosure.

FIGS. 3 and 4 illustrate examples of abnormal driving situationsaccording to aspects of the present disclosure.

FIG. 5 is a diagram illustrating an example of a hardware implementationfor passenger interaction system for an autonomous vehicle according toaspects of the present disclosure.

FIG. 6 illustrates a flow diagram for controlling an autonomous vehicleaccording to aspects of the present disclosure.

DETAILED DESCRIPTION

The detailed description set forth below, in connection with theappended drawings, is intended as a description of variousconfigurations and is not intended to represent the only configurationsin which the concepts described herein may be practiced. The detaileddescription includes specific details for the purpose of providing athorough understanding of the various concepts. It will be apparent tothose skilled in the art, however, that these concepts may be practicedwithout these specific details. In some instances, well-known structuresand components are shown in block diagram form in order to avoidobscuring such concepts.

As discussed, autonomous vehicles perform actions based on developedrules. For example, the autonomous vehicle may follow a set of ruleswhen driving on a multilane road. In this example, the set of rules mayinclude a rule that establishes a minimum following distance betweenvehicles in a same lane. The set of rules may include another ruleestablishing a minimum gap for merging between two vehicles. Theaforementioned rules are examples of rules, the rules of the autonomousvehicle are not limited to the rules discussed above. Additionally, forclarity, the autonomous vehicle may be referred to as a vehicle.

Actions of the autonomous vehicle are constrained by the establishedrules. In theory, if the autonomous vehicle follows each rule, theautonomous vehicle should not be the cause of an accident. To preventaccidents, the rules are set to be conservative. Due to the conservativenature of the rules, in some situations, such as heavy trafficsituations, the rules may increase driving times.

As an example, one rule establishes a minimum gap for merging. When thevehicle is in heavy traffic, the minimum gap may not be present.Therefore, the vehicle may wait for a prolonged period of time beforemerging. The prolonged wait period may be inconvenient to the vehicle'soccupant (e.g., passenger). Additionally, failure to merge may cause atraffic backup because vehicles behind the current vehicle cannot mergeuntil the current vehicle merges.

When a vehicle is operated in a manual mode (e.g., operated by a humandriver), the human may communicate with other drivers. For example, whena manually operated vehicle is attempting to merge in a heavy trafficsituation, the human driver may motion (e.g., wave) to the other driversto ask them to yield. The yielding vehicle may create a gap that allowsthe manually operated vehicle to merge.

Aspects of the present disclosure are directed to improving autonomousdriving systems by prompting a passenger to interact with other driversin response to identifying an abnormal driving situation. Theinteraction is similar to an interaction between human drivers duringmanual operation. Additionally, the interaction does not compromisesafety.

FIG. 1 illustrates an example of a vehicle 100 (e.g., ego vehicle) in anenvironment 150 according to aspects of the present disclosure. In thepresent example, the vehicle 100 is an autonomous vehicle. As shown inFIG. 1 , the vehicle 100 may be traveling on a road 110. A first vehicle104 may be ahead of the vehicle 100 and a second vehicle 116 may beadjacent to the ego vehicle 100. In this example, the vehicle 100 mayinclude a 2D camera 108, such as a 2D RGB camera, and a LIDAR sensor106. Other sensors, such as RADAR and/or ultrasound, are alsocontemplated. Additionally, or alternatively, the vehicle 100 mayinclude one or more additional 2D cameras and/or LIDAR sensors. Forexample, the additional sensors may be side facing and/or rear facingsensors.

In one configuration, the 2D camera 108 captures a 2D image thatincludes objects in the 2D camera's 108 field of view 114. The LIDARsensor 106 may generate one or more output streams. The first outputstream may include a 3D cloud point of objects in a first field of view,such as a 360° field of view 112 (e.g., bird's eye view). The secondoutput stream 124 may include a 3D cloud point of objects in a secondfield of view, such as a forward facing field of view.

The 2D image captured by the 2D camera includes a 2D image of the firstvehicle 104, as the first vehicle 104 is in the 2D camera's 108 field ofview 114. As is known to those of skill in the art, a LIDAR sensor 106uses laser light to sense the shape, size, and position of objects in anenvironment. The LIDAR sensor 106 may vertically and horizontally scanthe environment. In the current example, the artificial neural network(e.g., autonomous driving system) of the vehicle 100 may extract heightand/or depth features from the first output stream. The autonomousdriving system of the vehicle 100 may also extract height and/or depthfeatures from the second output stream.

The information obtained from the sensors 106, 108 may be used toevaluate a driving situation (e.g., driving environment). In oneconfiguration, the autonomous driving system determines whether acurrent driving situation is an abnormal driving situation.Additionally, the autonomous driving system determines if breaking oneor more rules will overcome (e.g., mitigate) the abnormal drivingsituation. Overcoming the abnormal driving situation may reduce anoverall drive time, reduce resource use (e.g., fuel or battery use),prevent a collision caused by another vehicle, or generally improve adriving situation. If the passenger in the autonomous vehicle is unableor unwilling to perform the task (e.g., if they are asleep ordistracted), the vehicle continues to follow the rules.

As discussed, during vehicle operation, the autonomous driving systemmonitors the current driving situation to determine if the currentdriving situation is an abnormal situation. As an example, an abnormalsituation may be a deadlock situation where movement of the ego vehicleand one or more vehicles has been less than a distance threshold forpre-determined period of time. The deadlock situation is one example ofan abnormal situation, the abnormal situation is not limited to thedeadlock situation.

The deadlock situation detection accounts for vehicle movement when avehicle is waiting. For example, when a vehicle is waiting at a redlight or stop sign, the vehicle is not always stationary. As a driver'simpatience increases, the vehicle may move up closer and closer.Alternatively, a driver may be tired of pressing the brake, and a slightrelease of the break may cause the vehicle to move.

As such, the deadlock situation uses a distance threshold whendetermining if the vehicle is stagnant. That is, if the vehicle'smovement over a period of time is less than a distance threshold, thevehicle may be considered stagnant. When the ego vehicle and anothervehicle are stagnant, the situation may be considered a deadlocksituation. The distance threshold may be different based on the currentenvironment. For example, the distance threshold for a stop sign may bedifferent from a distance threshold for merging.

As discussed, a deadlock situation accounts for a wait time and adistance traveled. For example, an ego vehicle and another vehicle maywait at an intersection for a period of time that satisfies a timethreshold (e.g., five seconds). Additionally, the movement (e.g.,distance traveled) of the ego vehicle and another vehicle may be lessthan a threshold (e.g., two feet). In this example, the situation may bea deadlock situation. The deadlock situation may be mitigated byviolating one or more rules. Still, to maintain safety, the autonomousdriving system does not violate a rule until receiving confirmation thatthe user has communicated (e.g., negotiated, interacted, etc.) with oneor more other drivers.

FIG. 2A illustrates an example of an abnormal driving situationaccording to aspects of the present disclosure. As shown in FIG. 2A, avehicle 200 (e.g., merging vehicle 200) is attempting to merge on to alane 204 from an onramp 206. In the current example, the merging vehicle200 is operating in an autonomous mode and includes one or morepassengers 250 (see FIG. 2C). Additionally, in the example of FIG. 2A,the merging vehicle 200 includes one or more rules for merging. One ruleestablishes a minimum merging gap for the merging vehicle 200. Theminimum merging gap establishes a minimum distance between two vehiclesthat must be present in order for the merging vehicle 200 to mergebetween the two vehicles.

For illustrative purposes, in FIG. 2A, the gap 208 between a frontvehicle 220 and a rear vehicle 212 is less than the minimum merging gap.The minimum gap may be established based on the length of the mergingvehicle 200. For example, the minimum gap may be a sum of a length ofthe merging vehicle 200, a front buffer, and a rear buffer. As anexample, if the length of the merging vehicle 200 is 14′, the minimumgap may be 22′ (e.g., 14′ (length of the vehicle)+4′ (front buffer)+4′(rear buffer)).

The merging vehicle 200 may identify the gap 208 based on measurementsperformed by one or more sensors (not shown in FIG. 2A) of the mergingvehicle 200. The sensors may include LIDAR, RADAR, camera, or othertypes of sensors. After measuring the gap 208, the merging vehicle 200(e.g., the autonomous system of the merging vehicle 200) determines ifthe gap 208 satisfies the minimum merging gap.

In the present example, the gap 208 is less than the minimum merginggap. Therefore, the autonomous system will not allow the merging vehicle200 to merge between the front vehicle 220 and the rear vehicle 212. Inconventional systems, the merging vehicle 200 waits until a subsequentgap is equal to or greater than the minimum merging gap. Still, waitingfor a satisfactory gap may delay drive time.

In one configuration, after determining that the gap 208 does notsatisfy the minimum merging gap, the merging vehicle 200 determines ifthe current situation is an abnormal situation. For example, the mergingvehicle 200 may scan the lane 204 to measure traffic. Trafficinformation may also be obtained via wireless communications (e.g.,cellular, WiFi, etc.), such as communications with an Internet trafficsite or a traffic data center.

Based on the scanning and/or traffic information, the merging vehicle200 may determine that the traffic is greater than a threshold. If thetraffic is greater than a threshold, the merging vehicle 200 determinesthat the current driving situation is abnormal. That is, due to theincreased traffic, a satisfactory gap may not be present. As such, basedon training, the merging vehicle determines that the abnormal drivingsituation may be mitigated if the rear vehicle 212 yields to expand thegap 208. The expanded gap (not shown) may satisfy the minimum merginggap. Alternatively, the expanded gap may be less than the minimummerging gap and greater than the gap 208 that was present before therear vehicle 212 yielded, thereby improving safety.

As another example, the merging vehicle 200 may determine that adistance traveled for a period of time is less than a distancethreshold. Furthermore, the merging vehicle 200 may also determine thatthe rear vehicle's 112 distance traveled for a period of time is lessthan a distance threshold. Therefore, the current driving situation maybe a deadlock situation between the merging vehicle 200 and the rearvehicle 112. As discussed, the deadlock situation is one type ofabnormal driving situation. The travel distance may be determined fromone or more sensors, such as a positioning sensor, LIDAR, RADAR, and/orother sensors.

After determining that the abnormal driving situation may be mitigatedby expanding the gap 208, the merging vehicle 200 determines if the rearvehicle 212 is operated by a human driver (e.g., manual mode). Themerging vehicle may also determine if the rear vehicle is operated bythe human driver prior to determining that the abnormal drivingsituation may be mitigated. In one configuration, the merging vehicleuses the gap 208 (e.g., length of the gap 208) to determine if the rearvehicle 212 is operated by a human.

As discussed, autonomous vehicles operate based on rules. Some rules maybe standardized by the industry and/or set by a government entity. Oneof the rules establishes a minimum following distance, such that anautonomous vehicle should be at least the minimum following distancebehind a preceding vehicle. In one configuration, if the gap 208 is lessthan the minimum following distance, the merging vehicle 200 determinesthat the rear vehicle 212 is operating in a manual mode.

That is, the rear vehicle 212 would not break the minimum followingdistance if the rear vehicle 212 was operating in an autonomous mode.Aspects of the present disclosure are not limited to using the minimumdistance rule for determining whether the rear vehicle 212 is operatingin the manual mode. The autonomous system may use other types of ruleviolations to determine whether the rear vehicle 212 is operating in themanual mode. Other rule violations may include, for example, a vehicle'sspeed being greater than a threshold, a vehicle using high beam lightson a busy road, or another type of rule violation.

Because the minimum merging gap is dynamic (e.g., based on a length ofthe merging vehicle 200), the minimum following distance may bedifferent from the minimum merging gap. Therefore, the gap 208 may begreater than the minimum following distance and less than the minimummerging distance. For example, the minimum merging distance for a truckis greater than the minimum merging distance of a two-seat roadster.Thus, for the truck, the gap 208 may not satisfy the truck's minimummerging distance even if the gap 208 is greater than the minimumfollowing distance.

In addition to, or alternate from using a rule violation to determine ifthe rear vehicle 212 is being operated in the manual mode, the mergingvehicle 200 may use information obtained from one or more sensors todetermine if the rear vehicle 212 is operating in the manual mode. Forexample, the image capturing device may capture a series of images of acockpit of the rear vehicle 212. The series of images may capture ahuman actively operating the rear vehicle 212 (e.g., sitting behind thesteering wheel, moving the steering wheel, etc.). Based on the series ofimages, the autonomous system of the merging vehicle 200 may determinethat the rear vehicle 212 is operating in the manual mode by a humandriver.

In one configuration, the merging vehicle 200 communicates with the rearvehicle 212, via a communication channel (e.g., cellular, satellite,WiFi, etc.), to determine if the rear vehicle 212 is operating in themanual mode. For example, the merging vehicle 200 may ping the rearvehicle 212 to request information regarding the rear vehicle's 212operating mode. As another example, the rear vehicle 212 may broadcastits operating mode. Determining a vehicle's operating mode via acommunication channel may be performed in addition to, or alternatefrom, one or more of the other methods for determining the operatingmode.

In another configuration, the merging vehicle 200 assumes that all othervehicles 212, 220 are manually operated. In this configuration, themerging vehicle 200 does not perform a function for determining whetherthe rear vehicle 212 is manually operated. Rather, upon detecting theabnormal driving situation, the merging vehicle 200 prompts thepassenger to interact with the driver of the rear vehicle 212. If therear vehicle 212 is not operated by a human, the passenger may informthe autonomous driving system that the rear vehicle 212 is autonomouslyoperated.

According to aspects of the present disclosure, after determining thatthe rear vehicle 212 is not operated by a human, the merging vehicle 200may continue one or more of the processes discussed above until amanually operated vehicle or a satisfactory gap is identified. Uponidentifying the manually operated vehicle (e.g., human operatedvehicle), the merging vehicle 200 prompts a passenger to negotiate withthe driver of the manually operated vehicle. For simplicity, it will beassumed that the rear vehicle 212 is manually operated.

The prompt may be provided via one or more displays of the mergingvehicle, such as a dashboard display, a radio interface, aheads-up-display (HUD), etc. The prompt instructs the passenger tocommunicate with the driver of the rear vehicle 212. The instruction maybe a specific instruction, such as “REQUEST A YIELD,” based on thecurrent environment and/or abnormal situation. The instruction may alsoindicate a specific vehicle (e.g., rear vehicle 212) that is a targetfor negotiation. Alternatively, the instruction may be a generalinstruction, such as “NEGOTIATE WITH OTHER DRIVERS.”

FIG. 2B illustrates an example of a prompt 228 provided to a passengervia a display unit 222 defined in a center console 230 of a vehicle(e.g., the merging vehicle 200). As discussed, the display unit 222 maybe a dashboard display, a radio interface, a HUD, or another type ofdisplay. In the current example, the display unit 222 is a touchscreenintegrated with the merging vehicle 200. The display unit 222 provides aprompt 228, such as “REQUEST A YIELD,” in response to the abnormaldriving situation.

The display unit 222 may also identify a target vehicle 226 that is atarget for negotiation (e.g., driver interaction). The target vehicle226 may correspond to another vehicle in the deadlock situation with themerging vehicle 200 (e.g., ego vehicle). In this example, the targetvehicle 226 corresponds to the rear vehicle 212.

The passenger may provide feedback to the merging vehicle 200 via thedisplay unit 222, or another interface, if the passenger successfullynegotiated with the driver of the rear vehicle 212. For example, thepassenger may provide an input via a first input interface 224 (e.g.,button) to indicate the negotiation was successful. As another example,the passenger may provide an input via a second input interface 232(e.g., button) to indicate the negotiation was unsuccessful.

FIG. 2C illustrates an example of a passenger interaction according toaspects of the present disclosure. As shown in FIG. 2C, in response tothe prompt 228, the passenger 250 of the merging vehicle 200 attempts aninteraction with a driver 252 of the rear vehicle 212. As an example,the interaction may be a waving motion, or another type of interactionto request a yield.

In response to the passenger's 250 interaction, the driver 252 mayacknowledge the passenger's 250 interaction. For example, the driver 252may provide a hand wave to acknowledge the passenger's 250 interaction.The hand wave may be an indication of a successful negotiation.

As another example, (not shown in FIG. 2C), the driver 252 may notrespond to the passenger's 250 interaction. In yet another example, (notshown in FIG. 2C), the driver 252 may refuse the passenger's 250 requestfor a yield. For example, the driver 252 may shake her head, indicatinga refusal. The driver's 252 failure to response or a refusal may be anindication of an unsuccessful negotiation.

If the passenger indicates the negotiation was successful, the mergingvehicle 200 will begin to merge into the gap 208. During the merge, oneor more sensors of the merging vehicle 200 will continue to monitor thegap 208 for safety. If the passenger indicates the negotiation wasunsuccessful or if the passenger does not respond, the merging vehicle200 will continue to follow the rules. That is, the merging vehicle 200will not merge into the gap 208 and waits for a satisfactory gap or asuccessful negotiation with another driver.

FIG. 3 illustrates an example of a driving situation that may beimproved by breaking a rule according to aspects of the presentdisclosure. As shown in FIG. 3 , a vehicle 300 (e.g., autonomous vehicle300) is stopped at an intersection 304 due to the presence of a stopsign 302. The vehicle 300 would like to continue moving through theintersection 304. Due to one or more rules, the vehicle 300 may becautious when crossing the intersection 304. A set of vehicles 306 maymisinterpret the cautiousness as hesitance and may also wait at theintersection.

If a wait time at the stop sign 302 exceeds a time threshold and atravel distance of the vehicle 300 and one or more vehicles of the setof vehicles 306 is less than a distance threshold, the vehicle 300 maydetermine that the current driving situation is a deadlock situation(e.g., abnormal driving situation). The time threshold may be apredetermined acceptable time for waiting at a stop sign 302. Thevehicle 300 may assume that one or more vehicles of the set of vehicles306 are manually operated. The vehicle 300 may then prompt the passengerto negotiate with drivers of the set of vehicles 306. The prompt mayrequest the passenger to ask the drivers of the set of vehicles 306 toyield, such that the vehicle 300 may proceed through the intersection304.

If the passenger indicates the negotiations (e.g., interactions) weresuccessful, the vehicle 300 will proceed through the intersection 304.If the passenger indicates the negotiations were unsuccessful or if thepassenger does not respond, the vehicle 300 will continue to follow therules and wait at the stop sign 302. That is, the vehicle 300 will notproceed through the intersection 304. The vehicle 300 may wait for thepassenger to successfully negotiate with another set of vehicles or fora break in traffic at the intersection 304.

FIG. 4 illustrates an example of a deadlock driving situation accordingto aspects of the present disclosure. As shown in FIG. 4 , a precedingvehicle 400 has stopped in a lane 402. The preceding vehicle 400 (e.g.,stopped vehicle 400) may have stopped due to a mechanical failure or foranother reason. In this example, an autonomous vehicle 404 (e.g.,vehicle 404) is stuck behind the stopped vehicle 400 because thevehicle's 404 rules do not allow the vehicle 404 to cross the dividingline 408 to drive around the stopped vehicle 400. Other cars 406 behindthe vehicle 404 are crossing a dividing line 408 to drive around thestopped vehicle 400. For simplicity, only one other car 406 is shownbehind the vehicle 404.

If a wait time behind the stopped vehicle 400 exceeds a time thresholdand movement from the vehicle 404 and the stopped vehicle 400 is lessthan a distance threshold, the vehicle 404 may determine that thecurrent driving situation is a deadlock situation (e.g., abnormaldriving situation). Additionally, or alternatively, the vehicle 404 maynotice, via one or more sensors, the other cars 406 driving around thestopped vehicle 400. In response, the vehicle 404 may determine that thecurrent situation is an abnormal driving situation. The vehicle 404 mayalso determine that the current driving situation may be improved bydriving around the stopped vehicle 400.

As discussed, one or more rules may prevent the vehicle 404 fromcrossing the dividing line 408 to drive around the stopped vehicle 400.Therefore, after identifying the abnormal driving situation, the vehicle404 prompts the passenger for permission to cross the dividing line 408to drive around the stopped vehicle 400. If the passenger providespermission, the vehicle 404 will proceed to cross the dividing line 408to drive around the stopped vehicle 400. If the passenger deniespermission or if the passenger does not respond, the vehicle 404 willcontinue to wait behind the stopped vehicle 400.

If the passenger did not provide a response, the vehicle 404 mayintermittently request permission. For example, the passenger may besleeping and unaware of the abnormal situation. Therefore, the vehicle404 may continue to ask until permission is received, or the abnormalsituation is resolved (e.g., the stopped car 400 moves).

FIG. 5 is a diagram illustrating an example of a hardware implementationfor a passenger interaction system 500, according to aspects of thepresent disclosure. The passenger interaction system 500 may be acomponent of a vehicle, a robotic device, or another device. Forexample, as shown in FIG. 5 , the passenger interaction system 500 is acomponent of an autonomous vehicle 528. Aspects of the presentdisclosure are not limited to the autonomous vehicle 528, as otherdevices, such as a bus, boat, drone, or robot, are also contemplated forusing the passenger interaction system 500. The autonomous vehicle 528may be autonomous or semi-autonomous.

The passenger interaction system 500 may be implemented with a busarchitecture, represented generally by a bus 550. The bus 550 mayinclude any number of interconnecting buses and bridges depending on thespecific application of the passenger interaction system 500 and theoverall design constraints. The bus 550 links together various circuitsincluding one or more processors and/or hardware modules, represented bya processor 520, a communication module 522, a location module 518, asensor module 502, a locomotion module 526, a navigation module 524, acomputer-readable medium 514, a passenger interaction module 508, and apassenger interface 512. The bus 550 may also link various othercircuits such as timing sources, peripherals, voltage regulators, andpower management circuits, which are well known in the art, andtherefore, will not be described any further.

The passenger interaction system 500 includes a transceiver 516 coupledto the processor 520, the sensor module 502, the passenger interactionmodule 508, the passenger interface 512, the communication module 522,the location module 518, the locomotion module 526, the navigationmodule 524, and the computer-readable medium 514. The transceiver 516 iscoupled to an antenna 544. The transceiver 516 communicates with variousother devices over a transmission medium. For example, the transceiver516 may receive commands via transmissions from a user or a remotedevice. As another example, the transceiver 516 may transmit drivingstatistics and information from the passenger interaction module 508 toa server (not shown).

The passenger interaction system 500 includes the processor 520 coupledto the computer-readable medium 514. The processor 520 performsprocessing, including the execution of software stored on thecomputer-readable medium 514 providing functionality according to thedisclosure. The software, when executed by the processor 520, causes thepassenger interaction system 500 to perform the various functionsdescribed for a particular device, such as the autonomous vehicle 528,or any of the modules 502, 514, 516, 518, 520, 522, 524, 526. Thecomputer-readable medium 514 may also be used for storing data that ismanipulated by the processor 520 when executing the software.

The sensor module 502 may be used to obtain measurements via differentsensors, such as a first sensor 506 and a second sensor 504. The firstsensor 506 may be a vision sensor, such as a stereoscopic camera or ared-green-blue (RGB) camera, for capturing 2D images. The second sensor504 may be a ranging sensor, such as a light detection and ranging(LIDAR) sensor or a radio detection and ranging (RADAR) sensor. Ofcourse, aspects of the present disclosure are not limited to theaforementioned sensors as other types of sensors, such as, for example,thermal, sonar, and/or lasers are also contemplated for either of thesensors 504, 506. The measurements of the first sensor 506 and thesecond sensor 504 may be processed by one or more of the processor 520,the sensor module 502, the communication module 522, the location module518, the passenger interaction module 508, the locomotion module 526,and the navigation module 524, in conjunction with the computer-readablemedium 514, to implement the functionality described herein. In oneconfiguration, the data captured by the first sensor 506 and the secondsensor 504 may be transmitted to an external device via the transceiver516. The first sensor 506 and the second sensor 504 may be coupled tothe autonomous vehicle 528 or may be in communication with theautonomous vehicle 528.

The location module 518 may be used to determine a location of theautonomous vehicle 528. For example, the location module 518 may use aglobal positioning system (GPS) to determine the location of theautonomous vehicle 528. The communication module 522 may be used tofacilitate communications via the transceiver 516. For example, thecommunication module 522 may be configured to provide communicationcapabilities via different wireless protocols, such as WiFi, long termevolution (LTE), 4G, etc. The communication module 522 may also be usedto communicate with other components of the autonomous vehicle 528 thatare not modules of the passenger interaction system 500.

The locomotion module 526 may be used to facilitate locomotion of theautonomous vehicle 528. As an example, the locomotion module 526 maycontrol the movement of the wheels. As another example, the locomotionmodule 526 may be in communication with a power source of the autonomousvehicle 528, such as an engine or batteries. Of course, aspects of thepresent disclosure are not limited to providing locomotion via wheelsand are contemplated for other types of components for providinglocomotion, such as propellers, treads, fins, and/or jet engines.

The passenger interaction system 500 also includes the navigation module524 for planning a route or controlling the locomotion of the autonomousvehicle 528, via the locomotion module 526. The navigation module 524may be in communication with the passenger interaction module 508, thesensor module 502, the transceiver 516, the processor 520, thecommunication module 522, the location module 518, the locomotion module526, the navigation module 524, and the computer-readable medium 514. Inone configuration, the navigation module 524 overrides the user input.The modules may be software modules running in the processor 520,resident/stored in the computer-readable medium 514, one or morehardware modules coupled to the processor 520, or some combinationthereof.

According to aspects of the present disclosure, the passengerinteraction system 500 includes a passenger interaction module 508 incommunication with the navigation module 524, passenger interface 512,the sensor module 502, the transceiver 516, the processor 520, thecommunication module 522, the location module 518, the locomotion module526, and the computer-readable medium 514.

In one configuration, the passenger interaction module 508 controls thepassenger interface 512 to prompt the passenger to perform an actionand/or to provide input. For example, as shown in FIG. 5 , the passengerinterface 512 displays a message “INTERACT WITH OTHER DRIVER.” Based oninformation provided by the navigation module 524, the sensor module502, the transceiver 516, the processor 520, the communication module522, the location module 518, the locomotion module 526, and thecomputer-readable medium 514, the passenger interaction module 508 maydetermine that the current driving situation is an abnormal situation(e.g., a situation that may be improved by breaking one or more rules).The passenger interaction module 508 may also identify the specificrule(s) that may be broken to improve the current driving situation.

The passenger interface 512 may be an interactive interface fordisplaying information and receiving an input. For example, thepassenger interface 512 may generate an output to request the passengerto interact with one or more human drivers. The human drivers may beidentified by the passenger interaction module 508 based on informationprovided by the sensor module 502, the processor 520, and/or thecommunication module 522. The passenger interface 512 may also providean option for the passenger to indicate whether the interaction wassuccessful or unsuccessful. Additionally, or alternatively, thepassenger interaction module may provide an input requesting permissionto break one or more rules.

In response to an indication that the interaction was successful or oneor more rules may be broken, the passenger interaction module 508instructs the navigation module 524 to control the locomotion module 526to autonomously navigate the autonomous vehicle 528 to break one or moreof the rules to improve the current driving situation. A notificationsystem, such as a display screen on a dashboard, of the autonomousvehicle 528 may provide the instructions to the driver.

FIG. 6 illustrates a method 600 for controlling an autonomous vehicleaccording to an aspect of the present disclosure. At block 602, anautonomous vehicle system navigates the autonomous vehicle based on aset of rules. The set of rules include rules to prevent a collision withanother object. For example, the set of rules may include a minimumvelocity, maximum velocity, minimum merging distance, minimum followingdistance, and other rules.

At block 604, the autonomous vehicle system identifies an abnormality ina current driving situation. As an example, a deadlock situation may bean abnormality. The deadlock situation occurs when a velocity of theautonomous vehicle and another vehicle is less than a threshold for aperiod of time. That is, a deadlock situation may be detected when aspeed of the autonomous vehicle and another vehicle is less than athreshold, for a period of time. As another example, the deadlocksituation may be detected when a distance traveled by the autonomousvehicle and another vehicle is less than a threshold, for a period oftime.

In another example, excess traffic may be an abnormality. That is, theabnormality is detected when an amount of traffic exceeds a trafficthreshold. The amount of traffic may be determined via one or morevehicle sensors and/or via information provided from a traffic controlcenter.

At block 606, the autonomous vehicle system (e.g., passenger interactionsystem) prompts a passenger of the autonomous vehicle to interact with adriver of a first vehicle in response to identifying the abnormality.The passenger may be prompted via an interactive display of theautonomous vehicle (see FIG. 2B). For example, the interactive displaymay be a touchscreen integrated with a center console or dashboard ofthe autonomous vehicle.

In one configuration, the autonomous vehicle assumes all other vehiclesare manually operated (e.g., operated by a human driver). In anotherconfiguration, the autonomous vehicle system may determine that anothervehicle is operating in a manual mode based on the other vehicleviolating one or more rules of the set of rules. Additionally, oralternatively, the autonomous vehicle system may determine that anothervehicle is operating in a manual mode based on an input from one or moresensors of the autonomous vehicle.

At block 608, the autonomous vehicle system controls the autonomousvehicle to violate one or more rules of the set of rules in response toan indication of a successful interaction with the driver. Theindication of the successful interaction may be received via an inputreceived at the interactive display. In one configuration, theautonomous vehicle continues to follow the set of rules in response toan indication of an unsuccessful interaction with the driver or when aninput is not received after prompting the passenger. Additionally, theautonomous vehicle system controls the autonomous vehicle to follow theset of rules after violating one or more rules to overcome the abnormalsituation.

Based on the teachings, one skilled in the art should appreciate thatthe scope of the present disclosure is intended to cover any aspect ofthe present disclosure, whether implemented independently of or combinedwith any other aspect of the present disclosure. For example, anapparatus may be implemented or a method may be practiced using anynumber of the aspects set forth. In addition, the scope of the presentdisclosure is intended to cover such an apparatus or method practicedusing other structure, functionality, or structure and functionality inaddition to, or other than the various aspects of the present disclosureset forth. It should be understood that any aspect of the presentdisclosure may be embodied by one or more elements of a claim.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother aspects.

Although particular aspects are described herein, many variations andpermutations of these aspects fall within the scope of the presentdisclosure. Although some benefits and advantages of the preferredaspects are mentioned, the scope of the present disclosure is notintended to be limited to particular benefits, uses or objectives.Rather, aspects of the present disclosure are intended to be broadlyapplicable to different technologies, system configurations, networksand protocols, some of which are illustrated by way of example in thefigures and in the following description of the preferred aspects. Thedetailed description and drawings are merely illustrative of the presentdisclosure rather than limiting, the scope of the present disclosurebeing defined by the appended claims and equivalents thereof.

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Additionally, “determining” may include receiving (e.g., receivinginformation), accessing (e.g., accessing data in a memory) and the like.Furthermore, “determining” may include resolving, selecting, choosing,establishing, and the like.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover: a, b, c,a-b, a-c, b-c, and a-b-c.

The various illustrative logical blocks, modules and circuits describedin connection with the present disclosure may be implemented orperformed with a processor configured to perform the functions discussedin the present disclosure. The processor may be a neural networkprocessor, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array signal (FPGA)or other programmable logic device (PLD), discrete gate or transistorlogic, discrete hardware components or any combination thereof designedto perform the functions described herein. The processor may be amicroprocessor, controller, microcontroller, or state machine speciallyconfigured as described herein. A processor may also be implemented as acombination of computing devices, e.g., a combination of a DSP and amicroprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or such other specialconfiguration, as described herein.

The steps of a method or algorithm described in connection with thepresent disclosure may be embodied directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in storage or machine readable medium,including random access memory (RAM), read only memory (ROM), flashmemory, erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), registers, a hard disk,a removable disk, a CD-ROM or other optical disk storage, magnetic diskstorage or other magnetic storage devices, or any other medium that canbe used to carry or store desired program code in the form ofinstructions or data structures and that can be accessed by a computer.A software module may comprise a single instruction, or manyinstructions, and may be distributed over several different codesegments, among different programs, and across multiple storage media. Astorage medium may be coupled to a processor such that the processor canread information from, and write information to, the storage medium. Inthe alternative, the storage medium may be integral to the processor.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims.

The functions described may be implemented in hardware, software,firmware, or any combination thereof. If implemented in hardware, anexample hardware configuration may comprise a processing system in adevice. The processing system may be implemented with a busarchitecture. The bus may include any number of interconnecting busesand bridges depending on the specific application of the processingsystem and the overall design constraints. The bus may link togethervarious circuits including a processor, machine-readable media, and abus interface. The bus interface may be used to connect a networkadapter, among other things, to the processing system via the bus. Thenetwork adapter may be used to implement signal processing functions.For certain aspects, a user interface (e.g., keypad, display, mouse,joystick, etc.) may also be connected to the bus. The bus may also linkvarious other circuits such as timing sources, peripherals, voltageregulators, power management circuits, and the like, which are wellknown in the art, and therefore, will not be described any further.

The processor may be responsible for managing the bus and processing,including the execution of software stored on the machine-readablemedia. Software shall be construed to mean instructions, data, or anycombination thereof, whether referred to as software, firmware,middleware, microcode, hardware description language, or otherwise.

In a hardware implementation, the machine-readable media may be part ofthe processing system separate from the processor. However, as thoseskilled in the art will readily appreciate, the machine-readable media,or any portion thereof, may be external to the processing system. By wayof example, the machine-readable media may include a transmission line,a carrier wave modulated by data, and/or a computer product separatefrom the device, all which may be accessed by the processor through thebus interface. Alternatively, or in addition, the machine-readablemedia, or any portion thereof, may be integrated into the processor,such as the case may be with cache and/or specialized register files.Although the various components discussed may be described as having aspecific location, such as a local component, they may also beconfigured in various ways, such as certain components being configuredas part of a distributed computing system.

The processing system may be configured with one or more microprocessorsproviding the processor functionality and external memory providing atleast a portion of the machine-readable media, all linked together withother supporting circuitry through an external bus architecture.Alternatively, the processing system may comprise one or moreneuromorphic processors for implementing the neuron models and models ofneural systems described herein. As another alternative, the processingsystem may be implemented with an application specific integratedcircuit (ASIC) with the processor, the bus interface, the userinterface, supporting circuitry, and at least a portion of themachine-readable media integrated into a single chip, or with one ormore field programmable gate arrays (FPGAs), programmable logic devices(PLDs), controllers, state machines, gated logic, discrete hardwarecomponents, or any other suitable circuitry, or any combination ofcircuits that can perform the various functions described throughoutthis present disclosure. Those skilled in the art will recognize howbest to implement the described functionality for the processing systemdepending on the particular application and the overall designconstraints imposed on the overall system.

The machine-readable media may comprise a number of software modules.The software modules may include a transmission module and a receivingmodule. Each software module may reside in a single storage device or bedistributed across multiple storage devices. By way of example, asoftware module may be loaded into RAM from a hard drive when atriggering event occurs. During execution of the software module, theprocessor may load some of the instructions into cache to increaseaccess speed. One or more cache lines may then be loaded into a specialpurpose register file for execution by the processor. When referring tothe functionality of a software module below, it will be understood thatsuch functionality is implemented by the processor when executinginstructions from that software module. Furthermore, it should beappreciated that aspects of the present disclosure result inimprovements to the functioning of the processor, computer, machine, orother system implementing such aspects.

If implemented in software, the functions may be stored or transmittedover as one or more instructions or code on a computer-readable medium.Computer-readable media include both computer storage media andcommunication media including any storage medium that facilitatestransfer of a computer program from one place to another. Additionally,any connection is properly termed a computer-readable medium. Forexample, if the software is transmitted from a website, server, or otherremote source using a coaxial cable, fiber optic cable, twisted pair,digital subscriber line (DSL), or wireless technologies such as infrared(IR), radio, and microwave, then the coaxial cable, fiber optic cable,twisted pair, DSL, or wireless technologies such as infrared, radio, andmicrowave are included in the definition of medium. Disk and disc, asused herein, include compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk, and Blu-Ray® disc where disksusually reproduce data magnetically, while discs reproduce dataoptically with lasers. Thus, in some aspects computer-readable media maycomprise non-transitory computer-readable media (e.g., tangible media).In addition, for other aspects computer-readable media may comprisetransitory computer-readable media (e.g., a signal). Combinations of theabove should also be included within the scope of computer-readablemedia.

Thus, certain aspects may comprise a computer program product forperforming the operations presented herein. For example, such a computerprogram product may comprise a computer-readable medium havinginstructions stored (and/or encoded) thereon, the instructions beingexecutable by one or more processors to perform the operations describedherein. For certain aspects, the computer program product may includepackaging material.

Further, it should be appreciated that modules and/or other appropriatemeans for performing the methods and techniques described herein can bedownloaded and/or otherwise obtained by a user terminal and/or basestation as applicable. For example, such a device can be coupled to aserver to facilitate the transfer of means for performing the methodsdescribed herein. Alternatively, various methods described herein can beprovided via storage means, such that a user terminal and/or basestation can obtain the various methods upon coupling or providing thestorage means to the device. Moreover, any other suitable technique forproviding the methods and techniques described herein to a device can beutilized.

It is to be understood that the claims are not limited to the preciseconfiguration and components illustrated above. Various modifications,changes, and variations may be made in the arrangement, operation, anddetails of the methods and apparatus described above without departingfrom the scope of the claims.

What is claimed is:
 1. A method performed by an autonomous vehicle,comprising: identifying a condition preventing the autonomous vehiclefrom proceeding along an intended route; prompting a passenger of theautonomous vehicle to interact with a driver of a first vehicle inresponse to identifying the condition; receiving, from the passenger, aninput at an interface of the autonomous vehicle indicating a successfulinteraction or an unsuccessful interaction with the driver; andcontrolling the autonomous vehicle to proceed along the intended routebased on the input indicating the successful interaction with thedriver.
 2. The method of claim 1, wherein the condition comprises one ormore of an amount of traffic on the intended route exceeding a trafficthreshold or a distance traveled by the autonomous vehicle over a periodof time being less than a distance threshold.
 3. The method of claim 1,further comprising determining that the first vehicle is operating in amanual mode based on one or more of the first vehicle violating atraffic rule or an input from a sensor of the autonomous vehicle.
 4. Themethod of claim 1, further comprising maintaining the autonomous vehicleat a current location based on the input indicating the unsuccessfulinteraction with the driver or failing to receive the input afterprompting the passenger.
 5. The method of claim 1, wherein the conditionis mitigated based on the successful interaction.
 6. The method of claim1, wherein: the autonomous vehicle operates according to a set of rules;and the autonomous vehicle violates one or more rules of the set ofrules based on proceeding along the intended route after receiving theinput indicating the successful interaction with the driver.
 7. Themethod of claim 6, further comprising controlling the autonomous vehicleto follow the set of rules after violating the one or more rules.
 8. Anapparatus an autonomous vehicle, comprising: a processor; a memorycoupled with the processor; and instructions stored in the memory andoperable, when executed by the processor, to cause the apparatus to:identify a condition preventing the autonomous vehicle from proceedingalong an intended route; prompt a passenger of the autonomous vehicle tointeract with a driver of a first vehicle in response to identifying thecondition; receive, from the passenger, an input at an interface of theautonomous vehicle indicating a successful interaction or anunsuccessful interaction with the driver; and control the autonomousvehicle to proceed along the intended route based on the inputindicating the successful interaction with the driver.
 9. The apparatusof claim 8, wherein the condition comprises one or more of an amount oftraffic on the intended route exceeding a traffic threshold or adistance traveled by the autonomous vehicle over a period of time beingless than a distance threshold.
 10. The apparatus of claim 8, whereinexecution of the instructions further cause the apparatus to determinethat the first vehicle is operating in a manual mode based on one ormore of the first vehicle violating a traffic rule or an input from asensor of the autonomous vehicle.
 11. The apparatus of claim 8, whereinexecution of the instructions further cause the apparatus to maintainthe autonomous vehicle at a current location based on the inputindicating the unsuccessful interaction with the driver or failing toreceive the input after prompting the passenger.
 12. The apparatus ofclaim 8, wherein the condition is mitigated based on the successfulinteraction.
 13. The apparatus of claim 8, wherein: the autonomousvehicle operates according to a set of rules; and the autonomous vehicleviolates one or more rules of the set of rules based on proceeding alongthe intended route after receiving the input indicating the successfulinteraction with the driver.
 14. The apparatus of claim 13, whereinexecution of the instructions further cause the apparatus to control theautonomous vehicle to follow the set of rules after violating the one ormore rules.
 15. A non-transitory computer-readable medium having programcode recorded thereon at an autonomous vehicle, the program codeexecuted by a processor and comprising: program code to identify acondition preventing the autonomous vehicle from proceeding along anintended route; program code to prompt a passenger of the autonomousvehicle to interact with a driver of a first vehicle in response toidentifying the condition; program code to receive, from the passenger,an input at an interface of the autonomous vehicle indicating asuccessful interaction or an unsuccessful interaction with the driver;and program code to control the autonomous vehicle to proceed along theintended route based on the input indicating the successful interactionwith the driver.
 16. The non-transitory computer-readable medium ofclaim 15, wherein the condition comprises one or more of an amount oftraffic on the intended route exceeding a traffic threshold or adistance traveled by the autonomous vehicle over a period of time beingless than a distance threshold.
 17. The non-transitory computer-readablemedium of claim 15, wherein the program code further comprises programcode to determine that the first vehicle is operating in a manual modebased on one or more of the first vehicle violating a traffic rule or aninput from a sensor of the autonomous vehicle.
 18. The non-transitorycomputer-readable medium of claim 15, wherein the program code furthercomprises program code to maintain the autonomous vehicle at a currentlocation based on the input indicating the unsuccessful interaction withthe driver or failing to receive the input after prompting thepassenger.
 19. The non-transitory computer-readable medium of claim 15,wherein the condition is mitigated based on the successful interaction.20. The non-transitory computer-readable medium of claim 15, wherein:the autonomous vehicle operates according to a set of rules; and theautonomous vehicle violates one or more rules of the set of rules basedon proceeding along the intended route after receiving the inputindicating the successful interaction with the driver.